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Abstract 


A  question  currently  being  argued  throughout  the  appraisal  community  concerns  evaluation  of 
artifact  quality  during  Standard  CMMI  Appraisal  Method  for  Process  Improvement 
(SCAMPI SM)  appraisals.  Different,  conflicting  opinions  on  whether  evaluation  of  artifact  quality 
during  a  SCAMPI  is  appropriate  run  from  an  extremely  vigorous  “no”  to  an  equally  vigorous 
“yes!”  In  other  words,  some  lead  appraisers  and  teams  believe  artifact  quality  is  totally  outside  the 
scope  of  a  SCAMPI  appraisal.  Other  appraisers  and  teams  believe  that  evaluating  artifact  quality 
is  an  integral  part  of  the  SCAMPI. 

This  technical  note  examines  the  various  opinions  in  an  attempt  to  resolve  the  confusion. 

The  main  issue  appears  to  be  linked  to  word  definitions — specifically,  a  lack  of  agreement  among 
SCAMPI  lead  appraisers  about  what  “quality”  means  in  the  SCAMPI  context.  This  report  notes 
that  earlier  writings  on  the  subject  focus  on  the  problem  of  defining  the  attributes  of  artifact  qual¬ 
ity  in  appraisals.  Three  attributes — appropriate,  adequate,  and  reasonable — are  recognized 
throughout  the  community  as  applicable  to  the  quality  of  artifacts.  However,  these  attributes  are 
not  defined,  creating  the  opportunity  to  apply  subjective  rules  where  objectivity  is  critical.  With 
this  information  in  hand,  we  can  initiate  an  effort  to  provide  more  clear-cut,  succinct  guidance  on 
the  application  of  quality  attributes  to  appraisal  artifacts.  This  effort  will  enable  Lead  Appraisers 
and  team  members  to  inject  greater  objectivity  into  appraisals. 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 


V  |  CMU/SEI-2009-TN-021 


VI  I  CMU/SEI-2009-TN-021 


1  Introduction 


A  question  currently  being  argued  throughout  the  appraisal  community  concerns  evaluation  of 
artifact  quality  during  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPIsm) 
appraisals.  Different,  conflicting  opinions  on  whether  evaluation  of  artifact  quality  during  a 
SCAMPI  is  appropriate  run  from  an  extremely  vigorous  “no”  to  an  equally  vigorous  “yes!”  In 
other  words,  some  SCAMPI  Lead  Appraisers  SM  and  teams  believe  artifact  quality  is  totally  out¬ 
side  the  scope  of  a  SCAMPI  appraisal.  Other  appraisers  and  teams  believe  that  artifact  quality  is 
an  integral  part  of  the  SCAMPI. 

This  technical  note  examines  the  various  opinions  in  an  attempt  to  resolve  the  confusion.  We  seek 
an  answer  to  the  following  question: 

Following  the  current  SCAMPI  methodology,  can  and  should  the  Lead  Appraiser  and  ap¬ 
praisal  teams  judge  the  quality  of  artifacts  examined  during  the  appraisal? 

To  explore  the  question,  we  have  to  understand  several  aspects  surrounding  quality  and  associated 
terminology: 

•  What  do  we  mean  by  quality  with  respect  to  artifacts  examined  during  an  appraisal? 

•  What  quality  attributes  does  the  SCAMPI  Method  allow  to  be  applied  to  artifacts,  if  any? 

•  What  would  be  the  effect  of  applying  the  quality  attributes  to  artifacts  during  a  SCAMPI? 

We  postulate  that  the  issues  noted  above  stem  from  the  lack  of  precise  definitions  of  terms  in  ap¬ 
praisal  documentation  related  to  the  word  “quality.”  To  correct  this,  we  introduce,  in  Section  2, 
terms  that  establish  a  context  for  the  subsequent  discussions  of  quality.  In  Section  3  we  note  asser¬ 
tions  made  by  the  appraisal  community  that  reflect  some  differences  in  that  community’s  thinking 
about  what  appraisals  can  accomplish  in  determining  process  quality.  Section  4  examines  these 
opinions  in  light  of  the  current  appraisal  methodology  in  a  SCAMPI. 


SM  SCAMPI  and  SCAMPI  Lead  Appraiser  are  service  marks  of  Carnegie  Mellon  University. 
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2  Terminology 


In  this  section,  we  introduce  and  define  terminology  related  to  the  issue  of  artifact  evaluation, 
specifically  the  term  quality. 

2.1  Entities — Quality  of  What 

The  first  question  to  answer:  What  entity  are  we  seeking  to  evaluate  for  quality  during  an  ap¬ 
praisal?  Baker  identifies  several,  such  as  product,  process,  artifact,  activity,  and  service,  to  which 
we  can  apply  quality  evaluations  [Baker  2007].  Certainly  other  entities  exist  for  this  application; 
the  concept  of  quality  is  generally  applicable  to  almost  anything,  tangible  or  intangible.  Each  en¬ 
tity  may  have  very  different  quality  attributes  being  evaluated  depending  on  one’s  perspective.  An 
appraiser  must  first  identify  the  entity  or  entities  for  which  the  quality  is  to  be  evaluated — and, 
critically,  the  associated  quality  attributes  to  examine  to  establish  an  evaluation  approach. 

From  Baker  and  the  question  at  hand,  the  following  is  the  focus  of  our  discussions: 

•  Artifacts  that  make  up  the  direct  or  indirect  objective  evidence  are  the  entities  whose  quality 
we  wish  to  attempt  to  judge. 

•  The  SCAMPI  Method  Definition  Document  (MDD)  can  provide  a  list  of  the  needed  quality 
attributes  that  can  be  applied  to  the  artifacts  for  a  more  objective  quality  evaluation  than  can 
be  obtained  through  the  consensus  of  individual  teams. 

•  This  evaluation  approach  can  result  in  support  for  reporting  on  the  quality  of  the  artifacts  as 
part  of  the  appraisal.  (How  to  use  this  information  is  a  decision  left  to  the  organization  being 
appraised.) 

2.2  Quality  (What  Do  We  Mean  by  Quality?) 

Cooper  provides  an  operational  definition  of  quality: 

Quality  is  the  degree  to  which  an  object  (entity)  [e.g.,  process,  product,  or  service]  satisfies 
a  specified  set  of  attributes  or  requirements  [Cooper  2002], 

This  identifies  “specified  requirements”  as  the  quality  attributes  to  be  used.  The  attributes  or  re¬ 
quirements  are  to  be  specified  and  must  adhere  to  the  normal  rules  for  requirements  description 
such  as  being  consistent,  clearly  stated,  testable,  or  measurable  (quantitative  in  some  way). 

Significant  here  is  the  concept  of  completely  defining  the  quality  attributes  being  sought.  In  order 
to  understand  the  quality  of  an  entity,  the  context 1  for  the  quality  attributes  must  be  defined  and 
specified.  This  concept  is  embedded  in  the  following  statement: 

Just  as  beauty  is  in  the  eye  of  the  beholder,  so  is  quality  [Baker  2007]. 


1  Context  includes  criteria  to  define  quality  in  quantitative  terms  and  the  associated  measurements  of 
quality. 


2  |  CMU/SEI-2009-TN-021 


Consequently,  a  set  of  attributes  that  one  community  deems  important  as  a  measure  of  quality 
may  not  be  deemed  important  by  another  community.  Rather,  each  community  is  likely  to  have  its 
own  set  of  attributes  and  attribute  values  with  which  to  measure  quality.  This  point  becomes  criti¬ 
cal  when  organizations  are  asked  to  establish  baselines  and  prediction  models  used  to  help  man¬ 
age  statistically  the  product’s  quality  and  process  performance  during  development. 

Overall  we  must  describe  the  entire  context  of  the  quality  we  are  looking  for  to  be  able  to  measure 
or  otherwise  judge  the  quality  of  an  entity. 

We  explore  these  quality  concepts  next  in  the  context  of  appraisals. 
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3  A  Review  of  Appraisers’ Opinions 


The  following  statements  summarize  opinions  of  the  appraisal  community  on  the  subject  of 
whether  an  appraisal  evaluates  artifact  quality. 

Some  Lead  Appraisers  and  teams  assert  the  following: 

An  appraisal  team  does  indeed  evaluate  the  quality  of  an  artifact.  Since  the  CMMI  practices 
describe  the  characteristics  of  effective  processes,  artifacts  that  address  implementation  of 
these  practices  are,  by  definition,  effective.  Since  effectiveness  is  a  quality  attribute,  the  arti¬ 
fact  is  subject  to  an  evaluation  of  quality. 

Other  Lead  Appraisers  and  teams  profess  the  following  approach: 

Quality  attributes  such  as  effectiveness  are  resented  for  practice  execution.  If  quality  attrib¬ 
utes  apply  to  process  and  practice  execution,  and  the  artifact  is  entirely  separate  from  prac¬ 
tice  execution,  then  there  are  no  quality  attributes  that  can  be  applied  to  the  artifacts. 

Still  other  Lead  Appraisers  and  teams  assert  the  following  view: 

Sooner  or  later  the  appraisal  team  must  establish  a  consensus  view  on  what  is  reasonable  to 
expect,  since  it  is  only  the  presence  of  that  consensus  view  that  permits  a  determination  of 
practice  implementation  to  be  made.  Artifacts  are  typically  judged  for  “adequacy  ”  as  an  in¬ 
dication  of  their  quality,  where  “adequacy”  depends  on  the  experience  of  the  team. 

These  statements  illustrate  some  of  the  confusion  surrounding  evaluation  of  artifact  quality. 


4  |  CMU/SEI-2009-TN-021 


4  Judging  Artifact  Quality  during  an  Appraisal 


4.1  Artifacts 

Appraisals  require  examination  of  artifacts  because  artifacts  are  considered  evidence  of  model 
practice  implementation  of  a  process.  Three  terms  are  part  of  the  appraisal  community’s  percep¬ 
tion  of  how  to  characterize  these  artifacts:  appropriate,  adequate,  and  reasonable.  As  such,  they 
may  be  considered  the  quality  attributes  of  the  artifacts  the  appraisal  examines. 

An  artifact  is  typically  considered  “appropriate”  if  it  is  applicable  to  the  practice  it  represents. 
Thus,  a  requirements  specification  would  not  be  appropriate  to  represent  the  output  of  the  practice 
“develop  a  project  plan.” 

Of  the  three  terms,  only  “adequacy”  is  further  elaborated  in  the  SCAMPI  MDD  as  “adequate  to 
substantiate  the  implementation  of  a  practice”  [MDD  2007]. 

However,  there  does  not  seem  to  be  any  further  definition  of  each  of  these  terms  in  the  MDD.  The 
presumption  by  many  is  that  the  appraisal  team  would  apply  their  experience  to  help  determine 
the  definition  and  be  able  to  judge  the  quality  of  the  artifact. 

Thus,  evaluation  of  an  artifact’s  “adequacy”  depends  on  the  experience  of  the  team,  and  as  as¬ 
serted,  “Beauty  is  in  the  eye  of  the  beholder.” 

Lastly,  the  team  might  be  able  to  define  “ reasonableness ”  of  the  artifact.  But  that  takes  us  again 
into  subjective  evaluation  of  artifacts.  This  is  similar  to  the  above  evaluation  of  artifact  adequacy 
being  dependent  on  the  experience  of  the  appraisal  team. 

Definitions  of  quality  attributes  in  the  CMMI  "’-DEV  glossary  do  not  provide  relief  or  additional 
insight.  For  "adequate”  the  glossary  states  the  following: 

This  word  is  used  so  that  you  can  interpret  goals  and  practices  in  light  of  your  organiza¬ 
tion  ’s  business  objectives.  When  using  any  CMMI  model,  you  must  interpret  the  practices  so 
that  they  work  for  your  organization.  This  term  is  used  in  the  goals  and  practices  where  cer¬ 
tain  activities  may  not  be  done  all  the  time.  (See  also  “appropriate”  and  “as  needed.  ”) 

The  terms  “appropriate”  and  “as  needed”  in  the  CMMI  glossary  simply  repeat  the  above  state¬ 
ment  and  are  therefore  treated  as  synonyms  for  adequate.  The  statement  itself  is  not  a  definition  in 
the  usual  sense,  but  it  does  illustrate  that  an  organization  can  or  should  use  the  terms  relative  to 
the  practices  it  implements.  For  example,  if  an  organization  defines  an  artifact  as  being  adequate 
relative  to  its  business  goals,  can  the  appraisal  team  judge  the  artifact  otherwise?  (Since  “appro¬ 
priate”  and  “as  needed”  are  used  synonymously,  this  fact  could  also  be  a  confusion  factor.) 


CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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4.2  Simple  Example 


A  simple  analogy  is  to  apply  the  three  terms  to  the  following  scenario.  We  will  use  the  following 
practice  as  an  example: 

Have  a  substantial  breakfast  before  9  a.m. 

During  the  appraisal,  the  team  identified  an  artifact  that  shows  that  the  organization  has  a  substan¬ 
tial  dinner  after  4  p.m.  This  artifact  is  not  considered  appropriate  because  it  does  not  relate  to  the 
practice.  Further  it  does  not  substantiate  implementation  of  the  practice,  so  it  is  not  adequate.  The 
quality  of  this  artifact’s  attributes  is  considered  substandard. 

Therefore,  we  may  conclude  that  the  artifact  is  neither  appropriate  nor  adequate. 

Being  a  conscientious  group,  the  appraisal  team  continues  to  look  for  and  finds  another  artifact  to 
address  the  breakfast  requirement.  It  turns  out  to  be  an  empty  glass  that  once  held  milk,  and  the 
organization  confirms  that  its  employee  breakfast  consists  only  of  milk.  The  artifact  is  appropriate 
because  it  addresses  breakfast;  and  it  is  adequate  to  substantiate  the  implementation  of  the  prac¬ 
tice.  But  is  it  reasonable  to  accept  a  glass  of  milk  as  the  sole  ingredient  for  a  substantial  breakfast? 
Who  will  determine  this? 

Using  the  team’s  collective  experience  and  knowledge  of  the  ingredients  of  a  substantial  break¬ 
fast,  the  decision  is  likely  to  be  that  this  artifact  cannot  be  considered  reasonable  as  proof  of  a 
substantial  breakfast. 

Note  that  our  practice  example  can  be  translated  any  number  of  ways  depending  on  culture  and 
common  practice.  Likewise,  organizations  can  implement  practices  any  number  of  ways,  based  on 
business  objectives.  If  the  organization  states  that  a  glass  of  milk  is  its  definition  of  a  substantial 
breakfast,  what  will  the  appraisal  team  decide?  Without  a  quantitative  basis  to  explain  this  and 
any  other  term,  there  is  no  way  to  provide  objective  basis  for  team  judgment  and  defense  of  team 
judgment.  There  is  an  argument  that  if  the  organization  uses  the  artifact,  then  it  is  deemed  reason¬ 
able.  However,  appraisal  teams  must  be  careful  that  the  “use”  is  then  reasonable  to  ensure  the  or¬ 
ganization  doesn’t  just  state  that  the  artifact  is  used. 

Of  course,  even  for  this  example  there  are  other  scenarios  surrounding  the  artifacts  that  we  could 
pursue. 
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5  Conclusions 


Let’s  consider  the  original  question:  Can  and  should  an  appraisal  team  evaluate  the  quality  of  arti¬ 
facts  discovered  during  SCAMPI  appraisals?  We  believe  the  answer  is  a  qualified  yes.  There  are 
three  attributes — appropriate,  adequate,  and  reasonable — that  are  recognized  throughout  the 
community  as  appropriate  to  apply  to  artifact  quality.  These  three  attributes  are  repeated  through¬ 
out  the  documentation  for  conducting  the  SCAMPI  family  of  appraisals  (A,  B,  and  C),  as  well  as 
throughout  the  CMMI  model  itself. 

However,  a  close  review  of  documents  that  guide  appraisal  teams  shows  that  these  quality  attrib¬ 
utes  have  been  designated  but  never  clearly  defined.  The  CMMI-DEV  glossary  uses  the  same 
definition  to  define  the  terms  adequate,  appropriate  and  a  third  term — “as  needed.”  This  glossary 
does  not  include  the  term  reasonable,  so  the  Webster  Dictionary  definition  of  the  term  takes 
precedence,  according  to  CMMI  rules  for  definitions.  That  definition  of  “reasonable”  is  (a)  “being 
in  accordance  with  reason  (a  reasonable  theory)  and  (b)  not  extreme  or  excessive.2  Applying  this 
definition  to  our  earlier  breakfast  example,  we  can  say  that  it  is  not  reasonable  to  define  a  glass  of 
milk  as  a  substantial  breakfast;  and  therefore,  the  artifact  does  not  support  the  quality  attribute  of 
being  reasonable. 

Unfortunately,  we  are  still  left  without  a  definitive  explanation  for  the  terms  “adequate”  and  “ap¬ 
propriate.”  There  have  been  indications  in  some  of  the  literature  that  this  lack  of  explanation  was 
a  conscious  decision  to  allow  the  terms  to  apply  to  a  variety  of  environments  and  a  variety  of  ap¬ 
proaches.  However,  the  lack  of  definition  creates  the  opportunity  to  apply  subjective  rules  where 
objectivity  is  critical. 

We  recommend  that  further  research  be  conducted  to  determine  the  requirements  used  by  the 
community  to  define  artifact  quality  attributes.  We  also  suggest  that  a  review  of  the  literature  may 
also  provide  material  on  commonly  held  definitions  for  these  terms  in  the  context  of  a  SCAMPI 
appraisal.  With  this  information  in  hand,  we  can  initiate  an  effort  to  provide  more  clear-cut  and 
succinct  guidance  on  the  application  of  quality  attributes  to  appraisal  artifacts.  The  results  of  this 
effort  will  enable  Lead  Appraisers  and  team  members  to  inject  greater  objectivity  into  the  ap¬ 
praisal  process  and  to  build  a  firmer  foundation  for  evaluating  the  quality  of  artifacts  that  they 
must  review. 


2  Definition  from  Merriam-Webster’s  Online  Dictionary  (http://www.merriam- 
webster.com/dictionary/reasonable) 
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